public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v21 1/7] Built-in compression method 123+ messages / 4 participants [nested] [flat]
* [PATCH v21 1/7] Built-in compression method @ 2020-09-08 09:54 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2020-09-08 09:54 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 99 +++++++ configure.ac | 22 ++ contrib/test_decoding/expected/ddl.out | 50 ++-- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 ++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 72 +++-- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++-- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 147 ++++++++++ .../access/compression/compress_pglz.c | 131 +++++++++ .../access/compression/compressamapi.c | 61 +++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 6 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 26 +- src/backend/commands/createas.c | 7 + src/backend/commands/matview.c | 7 + src/backend/commands/tablecmds.c | 101 ++++++- src/backend/executor/nodeModifyTable.c | 83 ++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 8 + .../replication/logical/reorderbuffer.c | 1 + src/backend/utils/adt/pg_upgrade_support.c | 10 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 46 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 46 +++- src/bin/pg_dump/pg_dump.h | 2 + src/bin/psql/describe.c | 42 ++- src/include/access/compressamapi.h | 65 +++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 22 +- src/include/catalog/binary_upgrade.h | 2 + src/include/catalog/pg_am.dat | 7 +- src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 10 +- src/include/catalog/pg_proc.dat | 24 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 3 +- src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/alter_table.out | 10 +- src/test/regress/expected/compression.out | 176 ++++++++++++ src/test/regress/expected/compression_1.out | 171 ++++++++++++ src/test/regress/expected/copy2.out | 8 +- src/test/regress/expected/create_table.out | 142 +++++----- .../regress/expected/create_table_like.out | 88 +++--- src/test/regress/expected/domain.out | 16 +- src/test/regress/expected/foreign_data.out | 258 +++++++++--------- src/test/regress/expected/identity.out | 16 +- src/test/regress/expected/inherit.out | 138 +++++----- src/test/regress/expected/insert.out | 118 ++++---- src/test/regress/expected/matview.out | 80 +++--- src/test/regress/expected/psql.out | 108 ++++---- src/test/regress/expected/publication.out | 40 +-- .../regress/expected/replica_identity.out | 14 +- src/test/regress/expected/rowsecurity.out | 16 +- src/test/regress/expected/rules.out | 30 +- src/test/regress/expected/stats_ext.out | 10 +- src/test/regress/expected/update.out | 16 +- src/test/regress/output/tablespace.source | 16 +- src/test/regress/parallel_schedule | 3 + src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 74 +++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 81 files changed, 2193 insertions(+), 668 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/backend/access/compression/compressamapi.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index e202697bbf..a7b4317b90 100755 --- a/configure +++ b/configure @@ -698,6 +698,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -865,6 +866,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld enable_largefile ' @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --without-lz4 do not use lz4 --with-gnu-ld assume the C compiler uses GNU ld [default=no] Some influential environment variables: @@ -8596,6 +8599,35 @@ fi +# +# lz4 +# + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + : + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12087,6 +12119,59 @@ fi fi +if test "$with_lz4" = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13290,6 +13375,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + +else + as_fn_error $? "lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index a5ad072ee4..ce29d6e97f 100644 --- a/configure.ac +++ b/configure.ac @@ -995,6 +995,13 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# lz4 +# +PGAC_ARG_BOOL(with, lz4, yes, + [do not use lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1182,6 +1189,14 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_LIB(lz4, LZ4_compress, [], + [AC_MSG_ERROR([lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1397,6 +1412,13 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADER(lz4.h, [], [AC_MSG_ERROR([lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..5bf614cc8a 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from its parent table, however a different compression method can be set + for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..d514c7f8c1 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This clause adds the compression method to a column. The Compression + method can be set from available compression methods. The built-in + methods are <literal>pglz</literal> and <literal>lz4</literal>. + If no compression method is specified, then compressible types will have + the default compression method <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..c90464053c 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,47 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_handler - get the compression handler routines * - * Decompress a compressed version of a varlena datum + * helper function for toast_decompress_datum and toast_decompress_datum_slice */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) { - struct varlena *result; + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +512,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); - - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..c4814ef911 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o compressamapi.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..e8b035d138 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,147 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2016-2020, PostgreSQL Global Development Group + * Portions Copyright (c) 1990-1993, Regents of the University of California + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "postgres.h" +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "fmgr.h" +#include "utils/builtins.h" + +#ifdef HAVE_LIBLZ4 +#include "lz4.h" +#endif + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value)); + tmp = (struct varlena *) palloc(max_size + TOAST_COMPRESS_HDRSZ); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + TOAST_COMPRESS_HDRSZ, + valsize, max_size); + if (len <= 0) + { + pfree(tmp); + elog(ERROR, "lz4: could not compress data"); + } + + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe_partial(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + slicelength, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..578be6f1dd --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,131 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Copyright (c) 2015-2018, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + TOAST_COMPRESS_HDRSZ); + + len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), + valsize, + TOAST_COMPRESS_RAWDATA(tmp), + NULL); + + if (len >= 0) + { + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + return tmp; + } + + pfree(tmp); + + return NULL; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + TOAST_COMPRESS_SIZE(value), + VARDATA(result), + TOAST_COMPRESS_RAWSIZE(value), true); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + VARDATA(result), + slicelength, false); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c new file mode 100644 index 0000000000..663102c8d2 --- /dev/null +++ b/src/backend/access/compression/compressamapi.c @@ -0,0 +1,61 @@ +/*------------------------------------------------------------------------- + * + * compression/compressamapi.c + * Functions for compression methods + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/access/compression/compressamapi.c + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/htup.h" +#include "access/htup_details.h" +#include "access/reloptions.h" +#include "access/table.h" +#include "utils/fmgroids.h" +#include "utils/syscache.h" + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index 009e215da1..44fa2ca7b4 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -181,6 +181,9 @@ my $C_COLLATION_OID = my $PG_CATALOG_NAMESPACE = Catalog::FindDefinedSymbolFromData($catalog_data{pg_namespace}, 'PG_CATALOG_NAMESPACE'); +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. @@ -808,6 +811,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b8cd35e995..e6040923dc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -347,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..668e43faed 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -34,6 +34,8 @@ static Oid lookup_am_handler_func(List *handler_name, char amtype); static const char *get_am_type_string(char amtype); +/* Set by pg_upgrade_support functions */ +Oid binary_upgrade_next_pg_am_oid = InvalidOid; /* * CreateAccessMethod @@ -83,7 +85,19 @@ CreateAccessMethod(CreateAmStmt *stmt) memset(values, 0, sizeof(values)); memset(nulls, false, sizeof(nulls)); - amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid)) + { + /* acoid should be found in some cases */ + if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId && + (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid)) + elog(ERROR, "could not link to built-in attribute compression"); + + amoid = binary_upgrade_next_pg_am_oid; + binary_upgrade_next_pg_am_oid = InvalidOid; + } + else + amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid); values[Anum_pg_am_amname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->amname)); @@ -175,6 +189,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..59690ce854 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -581,6 +581,13 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..41c66c9e61 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -486,6 +486,13 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..d7f4489c57 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,32 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * Get compression method Oid for the attribute. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + return InvalidOid; + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + return get_compression_am_oid(compression, false); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 5d90337498..684e836071 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,12 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_internals.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2039,78 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. + */ +void +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool isnull = false; + bool decompressed_any = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return; + + /* + * Loop for all the attributes in the tuple and check if any of the + * attribute is compressed in the source tuple and its compression method + * is not same as the target compression method then we need to decompress + * it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value = (struct varlena *) + DatumGetPointer(slot_getattr(slot, attnum, &isnull)); + + /* nothing to be done, if the value is null */ + if (isnull) + continue; + + /* nothing to be done. if it is not compressed */ + if (!VARATT_IS_COMPRESSED(new_value)) + continue; + + /* + * Get the compression method stored in the toast header and + * compare with the compression method of the target. + */ + if (targetTupDesc->attrs[i].attcompression != + CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value))) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the field in the source tuple then free + * the existing tuple. + */ + if (decompressed_any) + { + /* deform complete tuple before we clear the existing tuple */ + slot_getallattrs(slot); + + ExecClearTuple(slot); + slot->tts_nvalid = natts; + slot->tts_flags &= ~TTS_FLAG_EMPTY; + } +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2319,14 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress( + slot, resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index ba3ccc712c..3ea32323cd 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2932,6 +2932,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index a2ef853dc2..227bb07bbd 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2598,6 +2598,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6be19916fc..c6ba9e1b05 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3858,6 +3858,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 8392be6d44..187d3570e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2865,6 +2865,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 7574d545e0..355e273192 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -595,6 +595,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -630,9 +632,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3420,11 +3422,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3433,8 +3436,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3479,6 +3482,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3709,6 +3720,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15237,6 +15249,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15755,6 +15768,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..e2ac159aa2 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 5a62ab8bbc..43eee8a4b7 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -87,6 +87,7 @@ #include "access/detoast.h" #include "access/heapam.h" #include "access/rewriteheap.h" +#include "access/toast_internals.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog_internal.h" diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index a575c95079..f026104c01 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_am_oid = amoid; + PG_RETURN_VOID(); +} + Datum binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS) { diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..aff5b7e95e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_internals.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,50 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + char *compression; + int typlen; + struct varlena *varvalue; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + /* + * If it is not a varlena type or the attribute is not compressed then + * return NULL. + */ + if ((typlen != -1) || !VARATT_IS_COMPRESSED(value)) + PG_RETURN_NULL(); + + varvalue = (struct varlena *) DatumGetPointer(value); + + compression = + get_am_name(CompressionIdToOid(TOAST_COMPRESS_METHOD(varvalue))); + + PG_RETURN_TEXT_P(cstring_to_text(compression)); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 9d0056a569..5e168a3c04 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 39da742e32..becf565cdd 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -13016,6 +13035,11 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo) qamname = pg_strdup(fmtId(aminfo->dobj.name)); + if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION) + appendPQExpBuffer(q, + "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n", + aminfo->dobj.catId.oid); + appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname); switch (aminfo->amtype) @@ -15805,6 +15829,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15829,6 +15854,9 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15864,6 +15892,22 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Compression + * + * In binary-upgrade mode, compression is assigned by + * ALTER. Even if we're skipping compression the attribute + * will get default compression. It's the task for ALTER + * command to restore compression info. + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + has_non_default_compression) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 1290f9659b..53ece7abd7 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -327,6 +327,8 @@ typedef struct _tableInfo bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ + /* * Stuff computed only for dumpable tables. */ diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..4f82929fdb 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,27 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + bool mustfree = false; + const int trunclen = 100; + char *val = PQgetvalue(res, i, attcompression_col); + + /* truncate the options if they're too long */ + if (strlen(val) > trunclen + 3) + { + char *trunc = pg_malloc0(trunclen + 4); + strncpy(trunc, val, trunclen); + strncpy(trunc + trunclen, "...", 4); + + val = trunc; + mustfree = true; + } + + printTableAddCell(&cont, val, false, mustfree); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..8226ae0596 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,65 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Copyright (c) 2015-2017, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* access/compression/compressamapi.c */ +extern CompressionId CompressionOidToId(Oid cmoid); +extern Oid CompressionIdToOid(CompressionId cmid); + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..84ba303e7e 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,33 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; +#define RAWSIZEMASK (0x3FFFFFFFU) + /* * Utilities for manipulation of header information for compressed * toast entries. + * + * Since version 14 TOAST_COMPRESS_SET_RAWSIZE also marks compressed + * varlenas as custom compressed. Such varlenas will contain 0x02 (0b10) in + * two highest bits. */ #define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) +#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->info & RAWSIZEMASK) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> 30) #define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) #define TOAST_COMPRESS_RAWDATA(ptr) \ (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= RAWSIZEMASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << 30); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7ac5..9d65bae238 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid; + extern PGDLLIMPORT bool binary_upgrade_record_init_privs; #endif /* BINARY_UPGRADE_H */ diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..00362d9db3 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,10 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, - +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index 358f5aac8f..4a5c249ee9 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX(pg_am_oid_index, 2652, on pg_am using btree(oid oid_ops)); */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 059dec3647..a6cb1b4e5b 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -156,6 +156,14 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation */ Oid attcollation; + /* + * Oid of the compression method that will be used for compressing the value + * for this attribute. For compressible types this must always be a + * valid Oid irrespective of what is the current value of the attstorage. + * And for the incompressible types this must be InvalidOid. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -183,7 +191,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b5f52d4e4a..7458ed7d0d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7087,6 +7096,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7257,6 +7270,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, @@ -10749,6 +10769,10 @@ proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v', proparallel => 'r', prorettype => 'void', proargtypes => 'oid', prosrc => 'binary_upgrade_set_next_pg_authid_oid' }, +{ oid => '1137', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v', + proparallel => 'r', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_am_oid' }, { oid => '3591', descr => 'for use by pg_upgrade', proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f', provolatile => 'v', proparallel => 'u', prorettype => 'void', diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 56da2913bd..984002ba21 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -625,6 +625,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 758c3ca097..36d32f94aa 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -616,5 +616,6 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern void CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index caed683ba9..ded2ad9692 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -510,6 +510,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index dc2bb40926..232c9f1248 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 8c554e1f69..89bd2fd7a8 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -87,6 +87,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index f4d9f3b408..47f114b8a0 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..90c7454d2a 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -55,9 +55,9 @@ /* * struct varatt_external is a traditional "TOAST pointer", that is, the * information needed to fetch a Datum stored out-of-line in a TOAST table. - * The data is compressed if and only if va_extsize < va_rawsize - VARHDRSZ. - * This struct must not contain any padding, because we sometimes compare - * these pointers using memcmp. + * The data is compressed if and only if size in + * va_extinfo < va_rawsize - VARHDRSZ. This struct must not contain any + * padding, because we sometimes compare these pointers using memcmp. * * Note that this information is stored unaligned within actual tuples, so * you need to memcpy from the tuple into a local struct variable before @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -280,8 +281,11 @@ typedef struct #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & 0x3FFFFFFF) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> 30) /* Externally visible macros */ diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 0ce6ee4622..138df4be93 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -2196,11 +2196,11 @@ where oid = 'test_storage'::regclass; create index test_storage_idx on test_storage (b, a); alter table test_storage alter column a set storage external; \d+ test_storage - Table "public.test_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | external | | - b | integer | | | 0 | plain | | + Table "public.test_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | external | pglz | | + b | integer | | | 0 | plain | | | Indexes: "test_storage_idx" btree (b, a) diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..4dea269fc3 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,176 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..a50c7679bb --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,171 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: not built with lz4 support +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +-------- +(0 rows) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- +(0 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: not built with lz4 support +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- +(0 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out index c64f0719e7..1778c15c7a 100644 --- a/src/test/regress/expected/copy2.out +++ b/src/test/regress/expected/copy2.out @@ -511,10 +511,10 @@ begin end $$ language plpgsql immutable; alter table check_con_tbl add check (check_con_function(check_con_tbl.*)); \d+ check_con_tbl - Table "public.check_con_tbl" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | + Table "public.check_con_tbl" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | Check constraints: "check_con_tbl_check" CHECK (check_con_function(check_con_tbl.*)) diff --git a/src/test/regress/expected/create_table.out b/src/test/regress/expected/create_table.out index ed8c01b8de..3105115fee 100644 --- a/src/test/regress/expected/create_table.out +++ b/src/test/regress/expected/create_table.out @@ -498,11 +498,11 @@ Partition key: RANGE (a oid_ops, plusone(b), c, d COLLATE "C") Number of partitions: 0 \d+ partitioned2 - Partitioned table "public.partitioned2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Partitioned table "public.partitioned2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition key: RANGE (((a + 1)), substr(b, 1, 5)) Number of partitions: 0 @@ -511,11 +511,11 @@ ERROR: no partition of relation "partitioned2" found for row DETAIL: Partition key of the failing row contains ((a + 1), substr(b, 1, 5)) = (2, hello). CREATE TABLE part2_1 PARTITION OF partitioned2 FOR VALUES FROM (-1, 'aaaaa') TO (100, 'ccccc'); \d+ part2_1 - Table "public.part2_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Table "public.part2_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition of: partitioned2 FOR VALUES FROM ('-1', 'aaaaa') TO (100, 'ccccc') Partition constraint: (((a + 1) IS NOT NULL) AND (substr(b, 1, 5) IS NOT NULL) AND (((a + 1) > '-1'::integer) OR (((a + 1) = '-1'::integer) AND (substr(b, 1, 5) >= 'aaaaa'::text))) AND (((a + 1) < 100) OR (((a + 1) = 100) AND (substr(b, 1, 5) < 'ccccc'::text)))) @@ -552,11 +552,11 @@ select * from partitioned where partitioned = '(1,2)'::partitioned; (2 rows) \d+ partitioned1 - Table "public.partitioned1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.partitioned1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Partition of: partitioned FOR VALUES IN ('(1,2)') Partition constraint: (((partitioned1.*)::partitioned IS DISTINCT FROM NULL) AND ((partitioned1.*)::partitioned = '(1,2)'::partitioned)) @@ -609,10 +609,10 @@ CREATE TABLE part_p2 PARTITION OF list_parted FOR VALUES IN (2); CREATE TABLE part_p3 PARTITION OF list_parted FOR VALUES IN ((2+1)); CREATE TABLE part_null PARTITION OF list_parted FOR VALUES IN (null); \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Partitions: part_null FOR VALUES IN (NULL), part_p1 FOR VALUES IN (1), @@ -1049,21 +1049,21 @@ create table test_part_coll_cast2 partition of test_part_coll_posix for values f drop table test_part_coll_posix; -- Partition bound in describe output \d+ part_b - Table "public.part_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 1 | plain | | + Table "public.part_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 1 | plain | | | Partition of: parted FOR VALUES IN ('b') Partition constraint: ((a IS NOT NULL) AND (a = 'b'::text)) -- Both partition bound and partition key in describe output \d+ part_c - Partitioned table "public.part_c" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Partitioned table "public.part_c" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: parted FOR VALUES IN ('c') Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text)) Partition key: RANGE (b) @@ -1071,11 +1071,11 @@ Partitions: part_c_1_10 FOR VALUES FROM (1) TO (10) -- a level-2 partition's constraint will include the parent's expressions \d+ part_c_1_10 - Table "public.part_c_1_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Table "public.part_c_1_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: part_c FOR VALUES FROM (1) TO (10) Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text) AND (b IS NOT NULL) AND (b >= 1) AND (b < 10)) @@ -1104,46 +1104,46 @@ Number of partitions: 3 (Use \d+ to list them.) CREATE TABLE range_parted4 (a int, b int, c int) PARTITION BY RANGE (abs(a), abs(b), c); CREATE TABLE unbounded_range_part PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE); \d+ unbounded_range_part - Table "public.unbounded_range_part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.unbounded_range_part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL)) DROP TABLE unbounded_range_part; CREATE TABLE range_parted4_1 PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE); \d+ range_parted4_1 - Table "public.range_parted4_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND (abs(a) <= 1)) CREATE TABLE range_parted4_2 PARTITION OF range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE); \d+ range_parted4_2 - Table "public.range_parted4_2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 3) OR ((abs(a) = 3) AND (abs(b) > 4)) OR ((abs(a) = 3) AND (abs(b) = 4) AND (c >= 5))) AND ((abs(a) < 6) OR ((abs(a) = 6) AND (abs(b) <= 7)))) CREATE TABLE range_parted4_3 PARTITION OF range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE); \d+ range_parted4_3 - Table "public.range_parted4_3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 6) OR ((abs(a) = 6) AND (abs(b) >= 8))) AND (abs(a) <= 9)) @@ -1175,11 +1175,11 @@ SELECT obj_description('parted_col_comment'::regclass); (1 row) \d+ parted_col_comment - Partitioned table "public.parted_col_comment" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+--------------- - a | integer | | | | plain | | Partition key - b | text | | | | extended | | + Partitioned table "public.parted_col_comment" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+--------------- + a | integer | | | | plain | | | Partition key + b | text | | | | extended | pglz | | Partition key: LIST (a) Number of partitions: 0 @@ -1188,10 +1188,10 @@ DROP TABLE parted_col_comment; CREATE TABLE arrlp (a int[]) PARTITION BY LIST (a); CREATE TABLE arrlp12 PARTITION OF arrlp FOR VALUES IN ('{1}', '{2}'); \d+ arrlp12 - Table "public.arrlp12" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - a | integer[] | | | | extended | | + Table "public.arrlp12" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer[] | | | | extended | pglz | | Partition of: arrlp FOR VALUES IN ('{1}', '{2}') Partition constraint: ((a IS NOT NULL) AND ((a = '{1}'::integer[]) OR (a = '{2}'::integer[]))) @@ -1201,10 +1201,10 @@ create table boolspart (a bool) partition by list (a); create table boolspart_t partition of boolspart for values in (true); create table boolspart_f partition of boolspart for values in (false); \d+ boolspart - Partitioned table "public.boolspart" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | boolean | | | | plain | | + Partitioned table "public.boolspart" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | boolean | | | | plain | | | Partition key: LIST (a) Partitions: boolspart_f FOR VALUES IN (false), boolspart_t FOR VALUES IN (true) diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 10d17be23c..4da878ee1c 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -325,32 +325,32 @@ CREATE TABLE ctlt4 (a text, c text); ALTER TABLE ctlt4 ALTER COLUMN c SET STORAGE EXTERNAL; CREATE TABLE ctlt12_storage (LIKE ctlt1 INCLUDING STORAGE, LIKE ctlt2 INCLUDING STORAGE); \d+ ctlt12_storage - Table "public.ctlt12_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt12_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS); \d+ ctlt12_comments - Table "public.ctlt12_comments" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | extended | | A - b | text | | | | extended | | B - c | text | | | | extended | | C + Table "public.ctlt12_comments" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | extended | pglz | | A + b | text | | | | extended | pglz | | B + c | text | | | | extended | pglz | | C CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition NOTICE: merging column "b" with inherited definition NOTICE: merging constraint "ctlt1_a_check" with inherited definition \d+ ctlt1_inh - Table "public.ctlt1_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt1_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Inherits: ctlt1 @@ -364,12 +364,12 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt13_inh () INHERITS (ctlt1, ctlt3); NOTICE: merging multiple inherited definitions of column "a" \d+ ctlt13_inh - Table "public.ctlt13_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt13_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) "ctlt3_a_check" CHECK (length(a) < 5) @@ -380,12 +380,12 @@ Inherits: ctlt1, CREATE TABLE ctlt13_like (LIKE ctlt3 INCLUDING CONSTRAINTS INCLUDING INDEXES INCLUDING COMMENTS INCLUDING STORAGE) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition \d+ ctlt13_like - Table "public.ctlt13_like" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A3 - b | text | | | | extended | | - c | text | | | | external | | C + Table "public.ctlt13_like" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A3 + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | C Indexes: "ctlt13_like_expr_idx" btree ((a || c)) Check constraints: @@ -402,11 +402,11 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt_all (LIKE ctlt1 INCLUDING ALL); \d+ ctlt_all - Table "public.ctlt_all" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt_all" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt_all_pkey" PRIMARY KEY, btree (a) "ctlt_all_b_idx" btree (b) @@ -440,11 +440,11 @@ DETAIL: MAIN versus EXTENDED -- Check that LIKE isn't confused by a system catalog of the same name CREATE TABLE pg_attrdef (LIKE ctlt1 INCLUDING ALL); \d+ public.pg_attrdef - Table "public.pg_attrdef" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.pg_attrdef" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "pg_attrdef_pkey" PRIMARY KEY, btree (a) "pg_attrdef_b_idx" btree (b) @@ -461,11 +461,11 @@ CREATE SCHEMA ctl_schema; SET LOCAL search_path = ctl_schema, public; CREATE TABLE ctlt1 (LIKE ctlt1 INCLUDING ALL); \d+ ctlt1 - Table "ctl_schema.ctlt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "ctl_schema.ctlt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt1_pkey" PRIMARY KEY, btree (a) "ctlt1_b_idx" btree (b) diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out index 411d5c003e..6268b26562 100644 --- a/src/test/regress/expected/domain.out +++ b/src/test/regress/expected/domain.out @@ -266,10 +266,10 @@ explain (verbose, costs off) create rule silly as on delete to dcomptable do instead update dcomptable set d1.r = (d1).r - 1, d1.i = (d1).i + 1 where (d1).i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptype | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptype | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: @@ -403,10 +403,10 @@ create rule silly as on delete to dcomptable do instead update dcomptable set d1[1].r = d1[1].r - 1, d1[1].i = d1[1].i + 1 where d1[1].i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptypea | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptypea | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out index b9e25820bc..89466da603 100644 --- a/src/test/regress/expected/foreign_data.out +++ b/src/test/regress/expected/foreign_data.out @@ -1383,12 +1383,12 @@ CREATE TABLE fd_pt1 ( CREATE FOREIGN TABLE ft2 () INHERITS (fd_pt1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1404,12 +1404,12 @@ Inherits: fd_pt1 DROP FOREIGN TABLE ft2; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | CREATE FOREIGN TABLE ft2 ( c1 integer NOT NULL, @@ -1428,12 +1428,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1471,12 +1471,12 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1496,17 +1496,17 @@ ALTER TABLE fd_pt1 ADD COLUMN c6 integer; ALTER TABLE fd_pt1 ADD COLUMN c7 integer NOT NULL; ALTER TABLE fd_pt1 ADD COLUMN c8 integer; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1528,17 +1528,17 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1570,17 +1570,17 @@ ALTER TABLE fd_pt1 ALTER COLUMN c1 SET (n_distinct = 100); ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STATISTICS -1; ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STORAGE EXTERNAL; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | 0 | plain | | - c5 | integer | | | | plain | | - c6 | integer | | not null | | plain | | - c7 | integer | | | | plain | | - c8 | text | | | | external | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | 0 | plain | | | + c5 | integer | | | | plain | | | + c6 | integer | | not null | | plain | | | + c7 | integer | | | | plain | | | + c8 | text | | | | external | pglz | | Child tables: ft2 \d+ ft2 @@ -1608,12 +1608,12 @@ ALTER TABLE fd_pt1 DROP COLUMN c6; ALTER TABLE fd_pt1 DROP COLUMN c7; ALTER TABLE fd_pt1 DROP COLUMN c8; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1645,12 +1645,12 @@ SELECT relname, conname, contype, conislocal, coninhcount, connoinherit -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1692,12 +1692,12 @@ ALTER FOREIGN TABLE ft2 ADD CONSTRAINT fd_pt1chk2 CHECK (c2 <> ''); ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1723,12 +1723,12 @@ ALTER TABLE fd_pt1 DROP CONSTRAINT fd_pt1chk2 CASCADE; INSERT INTO fd_pt1 VALUES (1, 'fd_pt1'::text, '1994-01-01'::date); ALTER TABLE fd_pt1 ADD CONSTRAINT fd_pt1chk3 CHECK (c2 <> '') NOT VALID; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) NOT VALID Child tables: ft2 @@ -1750,12 +1750,12 @@ Inherits: fd_pt1 -- VALIDATE CONSTRAINT need do nothing on foreign tables ALTER TABLE fd_pt1 VALIDATE CONSTRAINT fd_pt1chk3; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) Child tables: ft2 @@ -1781,12 +1781,12 @@ ALTER TABLE fd_pt1 RENAME COLUMN c3 TO f3; -- changes name of a constraint recursively ALTER TABLE fd_pt1 RENAME CONSTRAINT fd_pt1chk3 TO f2_check; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | not null | | plain | 10000 | - f2 | text | | | | extended | | - f3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | not null | | plain | | 10000 | + f2 | text | | | | extended | pglz | | + f3 | date | | | | plain | | | Check constraints: "f2_check" CHECK (f2 <> ''::text) Child tables: ft2 @@ -1845,12 +1845,12 @@ CREATE TABLE fd_pt2 ( CREATE FOREIGN TABLE fd_pt2_1 PARTITION OF fd_pt2 FOR VALUES IN (1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1890,12 +1890,12 @@ ERROR: table "fd_pt2_1" contains column "c4" not found in parent "fd_pt2" DETAIL: The new partition may contain only the columns present in parent. DROP FOREIGN TABLE fd_pt2_1; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -1917,12 +1917,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') -- no attach partition validation occurs for foreign tables ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1945,12 +1945,12 @@ ERROR: cannot add column to a partition ALTER TABLE fd_pt2_1 ALTER c3 SET NOT NULL; ALTER TABLE fd_pt2_1 ADD CONSTRAINT p21chk CHECK (c2 <> ''); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1975,12 +1975,12 @@ ERROR: column "c1" is marked NOT NULL in parent table ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ALTER c2 SET NOT NULL; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -2003,12 +2003,12 @@ ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ADD CONSTRAINT fd_pt2chk1 CHECK (c1 > 0); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Check constraints: "fd_pt2chk1" CHECK (c1 > 0) diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index fbca0333a2..dc2af075ff 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -498,14 +498,14 @@ TABLE itest8; (2 rows) \d+ itest8 - Table "public.itest8" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+----------------------------------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | not null | generated always as identity | plain | | - f3 | integer | | not null | generated by default as identity | plain | | - f4 | bigint | | not null | generated always as identity | plain | | - f5 | bigint | | | | plain | | + Table "public.itest8" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+----------------------------------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | not null | generated always as identity | plain | | | + f3 | integer | | not null | generated by default as identity | plain | | | + f4 | bigint | | not null | generated always as identity | plain | | | + f5 | bigint | | | | plain | | | \d itest8_f2_seq Sequence "public.itest8_f2_seq" diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index 2b68aef654..7e709916f6 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -1052,13 +1052,13 @@ ALTER TABLE inhts RENAME aa TO aaa; -- to be failed ERROR: cannot rename inherited column "aa" ALTER TABLE inhts RENAME d TO dd; \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aa | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | - dd | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aa | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | + dd | integer | | | | plain | | | Inherits: inht1, inhs1 @@ -1071,14 +1071,14 @@ NOTICE: merging multiple inherited definitions of column "aa" NOTICE: merging multiple inherited definitions of column "b" ALTER TABLE inht1 RENAME aa TO aaa; \d+ inht4 - Table "public.inht4" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - y | integer | | | | plain | | - z | integer | | | | plain | | + Table "public.inht4" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + y | integer | | | | plain | | | + z | integer | | | | plain | | | Inherits: inht2, inht3 @@ -1088,14 +1088,14 @@ ALTER TABLE inht1 RENAME aaa TO aaaa; ALTER TABLE inht1 RENAME b TO bb; -- to be failed ERROR: cannot rename inherited column "b" \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - c | integer | | | | plain | | - d | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + c | integer | | | | plain | | | + d | integer | | | | plain | | | Inherits: inht2, inhs1 @@ -1135,33 +1135,33 @@ drop cascades to table inht4 CREATE TABLE test_constraints (id int, val1 varchar, val2 int, UNIQUE(val1, val2)); CREATE TABLE test_constraints_inh () INHERITS (test_constraints); \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Indexes: "test_constraints_val1_val2_key" UNIQUE CONSTRAINT, btree (val1, val2) Child tables: test_constraints_inh ALTER TABLE ONLY test_constraints DROP CONSTRAINT test_constraints_val1_val2_key; \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Child tables: test_constraints_inh \d+ test_constraints_inh - Table "public.test_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Inherits: test_constraints DROP TABLE test_constraints_inh; @@ -1172,27 +1172,27 @@ CREATE TABLE test_ex_constraints ( ); CREATE TABLE test_ex_constraints_inh () INHERITS (test_ex_constraints); \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Indexes: "test_ex_constraints_c_excl" EXCLUDE USING gist (c WITH &&) Child tables: test_ex_constraints_inh ALTER TABLE test_ex_constraints DROP CONSTRAINT test_ex_constraints_c_excl; \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Child tables: test_ex_constraints_inh \d+ test_ex_constraints_inh - Table "public.test_ex_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Inherits: test_ex_constraints DROP TABLE test_ex_constraints_inh; @@ -1202,37 +1202,37 @@ CREATE TABLE test_primary_constraints(id int PRIMARY KEY); CREATE TABLE test_foreign_constraints(id1 int REFERENCES test_primary_constraints(id)); CREATE TABLE test_foreign_constraints_inh () INHERITS (test_foreign_constraints); \d+ test_primary_constraints - Table "public.test_primary_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id | integer | | not null | | plain | | + Table "public.test_primary_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id | integer | | not null | | plain | | | Indexes: "test_primary_constraints_pkey" PRIMARY KEY, btree (id) Referenced by: TABLE "test_foreign_constraints" CONSTRAINT "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Foreign-key constraints: "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) Child tables: test_foreign_constraints_inh ALTER TABLE test_foreign_constraints DROP CONSTRAINT test_foreign_constraints_id1_fkey; \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Child tables: test_foreign_constraints_inh \d+ test_foreign_constraints_inh - Table "public.test_foreign_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Inherits: test_foreign_constraints DROP TABLE test_foreign_constraints_inh; diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index da50ee3b67..3cd4ad9e53 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -142,11 +142,11 @@ create rule irule3 as on insert to inserttest2 do also insert into inserttest (f4[1].if1, f4[1].if2[2]) select new.f1, new.f2; \d+ inserttest2 - Table "public.inserttest2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+----------+--------------+------------- - f1 | bigint | | | | plain | | - f2 | text | | | | extended | | + Table "public.inserttest2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | bigint | | | | plain | | | + f2 | text | | | | extended | pglz | | Rules: irule1 AS ON INSERT TO inserttest2 DO INSERT INTO inserttest (f3.if2[1], f3.if2[2]) @@ -448,11 +448,11 @@ from hash_parted order by part; -- test \d+ output on a table which has both partitioned and unpartitioned -- partitions \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: LIST (lower(a)) Partitions: part_aa_bb FOR VALUES IN ('aa', 'bb'), part_cc_dd FOR VALUES IN ('cc', 'dd'), @@ -470,10 +470,10 @@ drop table hash_parted; create table list_parted (a int) partition by list (a); create table part_default partition of list_parted default; \d+ part_default - Table "public.part_default" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "public.part_default" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: list_parted DEFAULT No partition constraint @@ -853,11 +853,11 @@ create table mcrparted6_common_ge_10 partition of mcrparted for values from ('co create table mcrparted7_gt_common_lt_d partition of mcrparted for values from ('common', maxvalue) to ('d', minvalue); create table mcrparted8_ge_d partition of mcrparted for values from ('d', minvalue) to (maxvalue, maxvalue); \d+ mcrparted - Partitioned table "public.mcrparted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.mcrparted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: RANGE (a, b) Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE), mcrparted2_b FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE), @@ -869,74 +869,74 @@ Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVAL mcrparted8_ge_d FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) \d+ mcrparted1_lt_b - Table "public.mcrparted1_lt_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted1_lt_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a < 'b'::text)) \d+ mcrparted2_b - Table "public.mcrparted2_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted2_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'b'::text) AND (a < 'c'::text)) \d+ mcrparted3_c_to_common - Table "public.mcrparted3_c_to_common" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted3_c_to_common" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('c', MINVALUE) TO ('common', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'c'::text) AND (a < 'common'::text)) \d+ mcrparted4_common_lt_0 - Table "public.mcrparted4_common_lt_0" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted4_common_lt_0" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MINVALUE) TO ('common', 0) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b < 0)) \d+ mcrparted5_common_0_to_10 - Table "public.mcrparted5_common_0_to_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted5_common_0_to_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 0) TO ('common', 10) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 0) AND (b < 10)) \d+ mcrparted6_common_ge_10 - Table "public.mcrparted6_common_ge_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted6_common_ge_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 10) TO ('common', MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 10)) \d+ mcrparted7_gt_common_lt_d - Table "public.mcrparted7_gt_common_lt_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted7_gt_common_lt_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MAXVALUE) TO ('d', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a > 'common'::text) AND (a < 'd'::text)) \d+ mcrparted8_ge_d - Table "public.mcrparted8_ge_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted8_ge_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'd'::text)) diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 4b3a2e0cb7..e078818496 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -94,11 +94,11 @@ CREATE MATERIALIZED VIEW mvtest_bb AS SELECT * FROM mvtest_tvvmv; CREATE INDEX mvtest_aa ON mvtest_bb (grandtot); -- check that plans seem reasonable \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -106,11 +106,11 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -118,19 +118,19 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvvm - Materialized view "public.mvtest_tvvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | View definition: SELECT mvtest_tvv.grandtot FROM mvtest_tvv; \d+ mvtest_bb - Materialized view "public.mvtest_bb" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_bb" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_aa" btree (grandtot) View definition: @@ -142,10 +142,10 @@ CREATE SCHEMA mvtest_mvschema; ALTER MATERIALIZED VIEW mvtest_tvm SET SCHEMA mvtest_mvschema; \d+ mvtest_tvm \d+ mvtest_tvmm - Materialized view "public.mvtest_tvmm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvmm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_tvmm_expr" UNIQUE, btree ((grandtot > 0::numeric)) "mvtest_tvmm_pred" UNIQUE, btree (grandtot) WHERE grandtot < 0::numeric @@ -155,11 +155,11 @@ View definition: SET search_path = mvtest_mvschema, public; \d+ mvtest_tvm - Materialized view "mvtest_mvschema.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "mvtest_mvschema.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -356,11 +356,11 @@ UNION ALL CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM mvtest_vt2 UNION ALL SELECT moo, 3*moo FROM mvtest_vt2; \d+ mv_test2 - Materialized view "public.mv_test2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - moo | integer | | | | plain | | - ?column? | integer | | | | plain | | + Materialized view "public.mv_test2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + moo | integer | | | | plain | | | + ?column? | integer | | | | plain | | | View definition: SELECT mvtest_vt2.moo, 2 * mvtest_vt2.moo @@ -493,14 +493,14 @@ drop cascades to materialized view mvtest_mv_v_4 CREATE MATERIALIZED VIEW mv_unspecified_types AS SELECT 42 as i, 42.5 as num, 'foo' as u, 'foo'::unknown as u2, null as n; \d+ mv_unspecified_types - Materialized view "public.mv_unspecified_types" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - i | integer | | | | plain | | - num | numeric | | | | main | | - u | text | | | | extended | | - u2 | text | | | | extended | | - n | text | | | | extended | | + Materialized view "public.mv_unspecified_types" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + i | integer | | | | plain | | | + num | numeric | | | | main | pglz | | + u | text | | | | extended | pglz | | + u2 | text | | | | extended | pglz | | + n | text | | | | extended | pglz | | View definition: SELECT 42 AS i, 42.5 AS num, diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..d6912f2109 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -2813,34 +2813,34 @@ CREATE TABLE tbl_heap(f1 int, f2 char(100)) using heap; CREATE VIEW view_heap_psql AS SELECT f1 from tbl_heap_psql; CREATE MATERIALIZED VIEW mat_view_heap_psql USING heap_psql AS SELECT f1 from tbl_heap_psql; \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \set HIDE_TABLEAM off \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap_psql \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap -- AM is displayed for tables, indexes and materialized views. @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out index 63d6ab7a4e..71388c197d 100644 --- a/src/test/regress/expected/publication.out +++ b/src/test/regress/expected/publication.out @@ -74,11 +74,11 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall (1 row) \d+ testpub_tbl2 - Table "public.testpub_tbl2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl2_pkey" PRIMARY KEY, btree (id) Publications: @@ -187,22 +187,22 @@ ALTER PUBLICATION testpub_default SET TABLE testpub_tbl1; ALTER PUBLICATION testpub_default ADD TABLE pub_test.testpub_nopk; ALTER PUBLICATION testpib_ins_trunct ADD TABLE pub_test.testpub_nopk, testpub_tbl1; \d+ pub_test.testpub_nopk - Table "pub_test.testpub_nopk" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - foo | integer | | | | plain | | - bar | integer | | | | plain | | + Table "pub_test.testpub_nopk" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + foo | integer | | | | plain | | | + bar | integer | | | | plain | | | Publications: "testpib_ins_trunct" "testpub_default" "testpub_fortbl" \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: @@ -224,11 +224,11 @@ ALTER PUBLICATION testpub_default DROP TABLE testpub_tbl1, pub_test.testpub_nopk ALTER PUBLICATION testpub_default DROP TABLE pub_test.testpub_nopk; ERROR: relation "testpub_nopk" is not part of the publication \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: diff --git a/src/test/regress/expected/replica_identity.out b/src/test/regress/expected/replica_identity.out index 79002197a7..20ac0012ef 100644 --- a/src/test/regress/expected/replica_identity.out +++ b/src/test/regress/expected/replica_identity.out @@ -153,13 +153,13 @@ SELECT relreplident FROM pg_class WHERE oid = 'test_replica_identity'::regclass; (1 row) \d+ test_replica_identity - Table "public.test_replica_identity" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | - keya | text | | not null | | extended | | - keyb | text | | not null | | extended | | - nonkey | text | | | | extended | | + Table "public.test_replica_identity" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | | + keya | text | | not null | | extended | pglz | | + keyb | text | | not null | | extended | pglz | | + nonkey | text | | | | extended | pglz | | Indexes: "test_replica_identity_pkey" PRIMARY KEY, btree (id) "test_replica_identity_expr" UNIQUE, btree (keya, keyb, (3)) diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index 9506aaef82..7ebdc30ded 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -938,14 +938,14 @@ CREATE POLICY pp1 ON part_document AS PERMISSIVE CREATE POLICY pp1r ON part_document AS RESTRICTIVE TO regress_rls_dave USING (cid < 55); \d+ part_document - Partitioned table "regress_rls_schema.part_document" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ----------+---------+-----------+----------+---------+----------+--------------+------------- - did | integer | | | | plain | | - cid | integer | | | | plain | | - dlevel | integer | | not null | | plain | | - dauthor | name | | | | plain | | - dtitle | text | | | | extended | | + Partitioned table "regress_rls_schema.part_document" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +---------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + did | integer | | | | plain | | | + cid | integer | | | | plain | | | + dlevel | integer | | not null | | plain | | | + dauthor | name | | | | plain | | | + dtitle | text | | | | extended | pglz | | Partition key: RANGE (cid) Policies: POLICY "pp1" diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 6173473de9..8546f02c1c 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3035,11 +3035,11 @@ select * from rules_log; create rule r3 as on delete to rules_src do notify rules_src_deletion; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3055,11 +3055,11 @@ Rules: create rule r4 as on insert to rules_src do instead insert into rules_log AS trgt SELECT NEW.* RETURNING trgt.f1, trgt.f2; create rule r5 as on update to rules_src do instead UPDATE rules_log AS trgt SET tag = 'updated' WHERE trgt.f1 = new.f1; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3086,11 +3086,11 @@ create rule rr as on update to rule_t1 do instead UPDATE rule_dest trgt SET (f2[1], f1, tag) = (SELECT new.f2, new.f1, 'updated'::varchar) WHERE trgt.f1 = new.f1 RETURNING new.*; \d+ rule_t1 - Table "public.rule_t1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rule_t1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: rr AS ON UPDATE TO rule_t1 DO INSTEAD UPDATE rule_dest trgt SET (f2[1], f1, tag) = ( SELECT new.f2, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 431b3fa3de..a36b7e6069 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -125,11 +125,11 @@ SELECT stxname, stxdndistinct, stxddependencies, stxdmcv ALTER STATISTICS ab1_a_b_stats SET STATISTICS -1; \d+ ab1 - Table "public.ab1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.ab1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Statistics objects: "public"."ab1_a_b_stats" (ndistinct, dependencies, mcv) ON a, b FROM ab1 diff --git a/src/test/regress/expected/update.out b/src/test/regress/expected/update.out index bf939d79f6..58e996fdee 100644 --- a/src/test/regress/expected/update.out +++ b/src/test/regress/expected/update.out @@ -711,14 +711,14 @@ DROP TRIGGER d15_insert_trig ON part_d_15_20; :init_range_parted; create table part_def partition of range_parted default; \d+ part_def - Table "public.part_def" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | bigint | | | | plain | | - c | numeric | | | | main | | - d | integer | | | | plain | | - e | character varying | | | | extended | | + Table "public.part_def" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | bigint | | | | plain | | | + c | numeric | | | | main | pglz | | + d | integer | | | | plain | | | + e | character varying | | | | extended | pglz | | Partition of: range_parted DEFAULT Partition constraint: (NOT ((a IS NOT NULL) AND (b IS NOT NULL) AND (((a = 'a'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'a'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'b'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '20'::bigint) AND (b < '30'::bigint))))) diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 94c5f023c6..458c297670 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -151,10 +151,10 @@ Indexes: Number of partitions: 2 (Use \d+ to list them.) \d+ testschema.part - Partitioned table "testschema.part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "testschema.part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Indexes: "part_a_idx" btree (a), tablespace "regress_tblspace" @@ -171,10 +171,10 @@ Indexes: "part1_a_idx" btree (a), tablespace "regress_tblspace" \d+ testschema.part1 - Table "testschema.part1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "testschema.part1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: testschema.part FOR VALUES IN (1) Partition constraint: ((a IS NOT NULL) AND (a = 1)) Indexes: diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e0e1ef71dd..2407ca945b 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -116,6 +116,9 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +#test toast compression +test: compression + # event triggers cannot run concurrently with any test that runs DDL test: event_trigger # this test also uses event triggers, so likewise run it by itself diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 081fce32e7..152c8cb5b7 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -203,3 +203,4 @@ test: explain test: event_trigger test: fast_default test: stats +test: compression diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..0750a6c80e --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,74 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + + + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2f28de0355..9ebfe19a34 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 721b230bf2..49a62d6e33 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0002-Add-default_toast_compression-GUC.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 02/12] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..3374012940 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b4ab0b88ad..0db2a7430f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b2457a6924..811dbc3741 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..e262ec5bcb 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1487710d59..4b3d34ae1a 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 943931f65d..ac3884b3a8 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --9Ek0hoCL9XbhcSqy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 123+ messages in thread
* psql: Option to use expanded mode for various meta-commands @ 2024-12-30 09:10 Dean Rasheed <[email protected]> 2024-12-30 15:48 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 0 siblings, 1 reply; 123+ messages in thread From: Dean Rasheed @ 2024-12-30 09:10 UTC (permalink / raw) To: PostgreSQL Hackers <[email protected]> The output from various psql meta-commands such as \df+ can be quite wide, making it hard to read, and there is another patch [1] that will make it even wider. The output is much more readable if expanded mode is used, but it's somewhat inconvenient to keep turning that on and off, if you don't want all other query output in expanded mode. Attached is a rough patch that adds 'x' as an option to all \d* psql meta-commands, which forces the output to be displayed in expanded mode. So, for example, you can type \dfx or \dfx+ (or equivalently \df+x) to show a list of functions in expanded mode. This applies to all \d* meta-commands except a bare \d. That limitation is necessary because \dx already means something else (list extensions). In the case of \d with a pattern, this limitation isn't really new, since that already always uses expanded=off for its output regardless of the current \x setting. However, it does mean that you can't use 'x' with \d without a pattern to get an expanded mode list of all relations. However, the limitation only applies as long as there isn't another character after the \d, so you can type \d+x and it will work the same as \dtvmsE+x, listing all relations in expanded mode. So in practice, this doesn't seem like such a bad limitation, since you're much less likely to need expanded mode with \d without '+'. There are a couple of other similar meta-commands (\l and \z) that I haven't looked at yet, but they should be easy to make work in the same way, if people think this is useful. Regards, Dean [1] https://www.postgresql.org/message-id/flat/[email protected].... Attachments: [text/x-patch] psql-meta-cmds-expanded-mode.patch (26.8K, ../../CAEZATCVXJk3KsmCncf7PAVbxdDAUDm3QzDgGT7mBYySWikuOYw@mail.gmail.com/2-psql-meta-cmds-expanded-mode.patch) download | inline diff: diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 72f3347e53..b2ba7776f2 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -864,6 +864,13 @@ testdb=> same line. </para> + <para> + Many of the meta-commands also allow <literal>x</literal> to be appended + as an option. This will cause the results to be displayed in expanded + mode, as if <literal>expanded=on</literal> were included in the list of + <literal>\pset</literal> options. See also <literal>\x</literal>. + </para> + <para> The following meta-commands are defined: @@ -1272,7 +1279,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-d"> - <term><literal>\d[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\d[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1321,12 +1328,24 @@ SELECT $1 \parse stmt1 foreign tables. This is purely a convenience measure. </para> + <para> + As with many other commands, if <literal>x</literal> is appended to + the command name, the results are displayed in expanded mode, but note + that this only applies when <command>\d</command> is used without a + <replaceable class="parameter">pattern</replaceable> argument, and + the <literal>x</literal> modifier cannot appear immediately after the + <command>\d</command> (because <command>\dx</command> is a different + command); the <literal>x</literal> modifier may only appear after a + <literal>S</literal> or <literal>+</literal> modifier. For example, + <command>\d+x</command> is equivalent to <command>\dtvmsE+x</command> + and will show a list of all relations in expanded mode. + </para> </note> </listitem> </varlistentry> <varlistentry id="app-psql-meta-command-da-lc"> - <term><literal>\da[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\da[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1342,7 +1361,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-da-uc"> - <term><literal>\dA[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dA[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1357,7 +1376,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dac"> <term> - <literal>\dAc[+] + <literal>\dAc[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">input-type-pattern</replaceable></link>]] </literal> @@ -1380,7 +1399,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-daf"> <term> - <literal>\dAf[+] + <literal>\dAf[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">input-type-pattern</replaceable></link>]] </literal> @@ -1403,7 +1422,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dao"> <term> - <literal>\dAo[+] + <literal>\dAo[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">operator-family-pattern</replaceable></link>]] </literal> @@ -1427,7 +1446,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dap"> <term> - <literal>\dAp[+] + <literal>\dAp[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">operator-family-pattern</replaceable></link>]] </literal> @@ -1449,7 +1468,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-db"> - <term><literal>\db[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\db[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1465,7 +1484,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dc-lc"> - <term><literal>\dc[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dc[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists conversions between character-set encodings. @@ -1483,7 +1502,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dconfig"> - <term><literal>\dconfig[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dconfig[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists server configuration parameters and their values. @@ -1502,7 +1521,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dc-uc"> - <term><literal>\dC[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dC[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists type casts. @@ -1517,7 +1536,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dd-lc"> - <term><literal>\dd[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dd[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Shows the descriptions of objects of type <literal>constraint</literal>, @@ -1546,7 +1565,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dd-uc"> - <term><literal>\dD[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dD[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists domains. If <replaceable @@ -1563,7 +1582,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-ddp"> - <term><literal>\ddp [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\ddp[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists default access privilege settings. An entry is shown for @@ -1585,12 +1604,12 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-de"> - <term><literal>\dE[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\di[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dm[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\ds[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dt[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dv[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dE[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\di[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dm[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\ds[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dt[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dv[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1618,7 +1637,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-des"> - <term><literal>\des[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\des[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign servers (mnemonic: <quote>external @@ -1634,7 +1653,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-det"> - <term><literal>\det[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\det[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign tables (mnemonic: <quote>external tables</quote>). @@ -1649,7 +1668,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-deu"> - <term><literal>\deu[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\deu[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists user mappings (mnemonic: <quote>external @@ -1672,7 +1691,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dew"> - <term><literal>\dew[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dew[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign-data wrappers (mnemonic: <quote>external @@ -1688,7 +1707,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-df-lc"> - <term><literal>\df[anptwS+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> ... ] ]</literal></term> + <term><literal>\df[anptwSx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> ... ] ]</literal></term> <listitem> <para> @@ -1722,7 +1741,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-df-uc"> - <term><literal>\dF[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dF[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search configurations. @@ -1736,7 +1755,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dfd"> - <term><literal>\dFd[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFd[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search dictionaries. @@ -1750,7 +1769,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dfp"> - <term><literal>\dFp[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFp[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search parsers. @@ -1764,7 +1783,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dft"> - <term><literal>\dFt[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFt[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search templates. @@ -1778,7 +1797,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dg"> - <term><literal>\dg[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dg[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists database roles. @@ -1798,7 +1817,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dl-lc"> - <term><literal>\dl[+]</literal></term> + <term><literal>\dl[x+]</literal></term> <listitem> <para> This is an alias for <command>\lo_list</command>, which shows a @@ -1811,7 +1830,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dl-uc"> - <term><literal>\dL[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dL[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists procedural languages. If <replaceable @@ -1828,7 +1847,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dn"> - <term><literal>\dn[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dn[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1845,7 +1864,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-do-lc"> - <term><literal>\do[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> [ <replaceable class="parameter">arg_pattern</replaceable> ] ] ]</literal></term> + <term><literal>\do[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> [ <replaceable class="parameter">arg_pattern</replaceable> ] ] ]</literal></term> <listitem> <para> Lists operators with their operand and result types. @@ -1870,7 +1889,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-do-uc"> - <term><literal>\dO[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dO[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists collations. @@ -1890,7 +1909,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dp-lc"> - <term><literal>\dp[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dp[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists tables, views and sequences with their @@ -1914,7 +1933,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dp-uc"> - <term><literal>\dP[itn+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dP[itnx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists partitioned relations. @@ -1947,7 +1966,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-drds"> - <term><literal>\drds [ <link linkend="app-psql-patterns"><replaceable class="parameter">role-pattern</replaceable></link> [ <link linkend="app-psql-patterns"><replaceable class="parameter">database-pattern</replaceable></link> ] ]</literal></term> + <term><literal>\drds[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">role-pattern</replaceable></link> [ <link linkend="app-psql-patterns"><replaceable class="parameter">database-pattern</replaceable></link> ] ]</literal></term> <listitem> <para> Lists defined configuration settings. These settings can be @@ -1970,7 +1989,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-drg"> - <term><literal>\drg[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\drg[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists information about each granted role membership, including @@ -1990,7 +2009,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-drp"> - <term><literal>\dRp[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dRp[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists replication publications. @@ -2004,7 +2023,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-drs"> - <term><literal>\dRs[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dRs[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists replication subscriptions. @@ -2018,7 +2037,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dt"> - <term><literal>\dT[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dT[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists data types. @@ -2035,7 +2054,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-du"> - <term><literal>\du[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\du[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists database roles. @@ -2054,7 +2073,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dx-lc"> - <term><literal>\dx[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dx[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists installed extensions. @@ -2068,7 +2087,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dx-uc"> - <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dX[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists extended statistics. @@ -2090,7 +2109,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dy"> - <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dy[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists event triggers. @@ -5386,6 +5405,30 @@ testdb=> <userinput>\df int*pl * bigint</userinput> </programlisting> </para> + <para> + Here, the <literal>+</literal> option is used to display additional + information about one of these functions, and <literal>x</literal> is used + to display the results in expanded mode: +<programlisting> +testdb=> <userinput>\df+x int4*pl * bigint</userinput> +List of functions +-[ RECORD 1 ]-------+----------------------------- +Schema | pg_catalog +Name | int48pl +Result data type | bigint +Argument data types | integer, bigint +Type | func +Volatility | immutable +Parallel | safe +Owner | pgdevel +Security | invoker +Access privileges | +Language | internal +Internal name | int48pl +Description | implementation of + operator +</programlisting> + </para> + <para> When suitable, query results can be shown in a crosstab representation with the <command>\crosstabview</command> command: diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c index cd16f27947..462b23118f 100644 --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -850,6 +850,7 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) char *pattern; bool show_verbose, show_system; + unsigned short int save_expanded; /* We don't do SQLID reduction on the pattern yet */ pattern = psql_scan_slash_option(scan_state, @@ -858,6 +859,16 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) show_verbose = strchr(cmd, '+') ? true : false; show_system = strchr(cmd, 'S') ? true : false; + /* + * The 'x' option turns expanded mode on for this command only. This + * is allowed in all \d* commands, except \d by itself, since \dx is a + * separate command, and so the 'x' option cannot appear immediately + * after the initial 'd'. + */ + save_expanded = pset.popt.topt.expanded; + if (strchr(&cmd[2], 'x')) + pset.popt.topt.expanded = 1; + switch (cmd[1]) { case '\0': @@ -873,13 +884,14 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) { char *pattern2 = NULL; - if (pattern && cmd[2] != '\0' && cmd[2] != '+') + if (pattern && cmd[2] != '\0' && cmd[2] != '+' && cmd[2] != 'x') pattern2 = psql_scan_slash_option(scan_state, OT_NORMAL, NULL, true); switch (cmd[2]) { case '\0': case '+': + case 'x': success = describeAccessMethods(pattern, show_verbose); break; case 'c': @@ -941,6 +953,7 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) case 'p': case 't': case 'w': + case 'x': success = exec_command_dfo(scan_state, cmd, pattern, show_verbose, show_system); break; @@ -981,6 +994,7 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) case 't': case 'i': case 'n': + case 'x': success = listPartitionedTables(&cmd[2], pattern, show_verbose); break; default: @@ -1041,6 +1055,7 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) { case '\0': case '+': + case 'x': success = listTSConfigs(pattern, show_verbose); break; case 'p': @@ -1093,6 +1108,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) status = PSQL_CMD_UNKNOWN; } + /* Restore original expanded mode */ + pset.popt.topt.expanded = save_expanded; + free(pattern); } else diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 2657abdc72..cbfb37e716 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -309,9 +309,9 @@ describeFunctions(const char *functypes, const char *func_pattern, /* No "Parallel" column before 9.6 */ static const bool translate_columns_pre_96[] = {false, false, false, false, true, true, false, true, false, false, false, false}; - if (strlen(functypes) != strspn(functypes, "anptwS+")) + if (strlen(functypes) != strspn(functypes, "anptwSx+")) { - pg_log_error("\\df only takes [anptwS+] as options"); + pg_log_error("\\df only takes [anptwSx+] as options"); return true; } ^ permalink raw reply [nested|flat] 123+ messages in thread
* Re: psql: Option to use expanded mode for various meta-commands 2024-12-30 09:10 psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> @ 2024-12-30 15:48 ` Greg Sabino Mullane <[email protected]> 2025-01-08 13:43 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 0 siblings, 1 reply; 123+ messages in thread From: Greg Sabino Mullane @ 2024-12-30 15:48 UTC (permalink / raw) To: Dean Rasheed <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> I like this, very useful. It's a shame about the conflict with \dx (lesson for the future: think extra carefully about option namings!). I am impressed that \dx \d \d+ \d+x and even \dxx all work as one might intuit with this patch. Cheers, Greg ^ permalink raw reply [nested|flat] 123+ messages in thread
* Re: psql: Option to use expanded mode for various meta-commands 2024-12-30 09:10 psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2024-12-30 15:48 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> @ 2025-01-08 13:43 ` Dean Rasheed <[email protected]> 2025-01-09 03:18 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 0 siblings, 1 reply; 123+ messages in thread From: Dean Rasheed @ 2025-01-08 13:43 UTC (permalink / raw) To: Greg Sabino Mullane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Mon, 30 Dec 2024 at 15:48, Greg Sabino Mullane <[email protected]> wrote: > > I like this, very useful. It's a shame about the conflict with \dx (lesson for the future: think extra carefully about option namings!). I am impressed that \dx \d \d+ \d+x and even \dxx all work as one might intuit with this patch. > Thanks for looking. Attached is a more complete patch, now with - trivial bug-fix for previous \d code, per cfbot - expanded mode support for \l and \z - updated psql help - a few representative regression test cases The majority of the patch is doc updates, which are somewhat tedious. Initially, I resisted documenting the 'x' option in the description of every command affected, but given the length of the list of commands in the HTML page, it's a long way from any given command to the top or bottom where 'x' is described in more detail. So in the end, I decided to just add a sentence to each command's description, keeping it as short as possible. Regards, Dean Attachments: [text/x-patch] psql-meta-cmds-expanded-mode-v2.patch (63.5K, ../../CAEZATCUyLVQ-Ara0=944ubJY3t0qCiGz7e1o0tgftue3g-jLew@mail.gmail.com/2-psql-meta-cmds-expanded-mode-v2.patch) download | inline diff: diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml new file mode 100644 index 72f3347..c48bcfb --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -865,6 +865,13 @@ testdb=> </para> <para> + Many of the meta-commands also allow <literal>x</literal> to be appended + as an option. This will cause the results to be displayed in expanded + mode, as if <literal>expanded=on</literal> were included in the list of + <literal>\pset</literal> options. See also <literal>\x</literal>. + </para> + + <para> The following meta-commands are defined: <variablelist> @@ -1272,7 +1279,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-d"> - <term><literal>\d[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\d[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1321,12 +1328,24 @@ SELECT $1 \parse stmt1 foreign tables. This is purely a convenience measure. </para> + <para> + As with many other commands, if <literal>x</literal> is appended to + the command name, the results are displayed in expanded mode, but note + that this only applies when <command>\d</command> is used without a + <replaceable class="parameter">pattern</replaceable> argument, and + the <literal>x</literal> modifier cannot appear immediately after the + <command>\d</command> (because <command>\dx</command> is a different + command); the <literal>x</literal> modifier may only appear after an + <literal>S</literal> or <literal>+</literal> modifier. For example, + <command>\d+x</command> is equivalent to <command>\dtvmsE+x</command> + and will show a list of all relations in expanded mode. + </para> </note> </listitem> </varlistentry> <varlistentry id="app-psql-meta-command-da-lc"> - <term><literal>\da[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\da[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1337,19 +1356,23 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> </listitem> </varlistentry> <varlistentry id="app-psql-meta-command-da-uc"> - <term><literal>\dA[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dA[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists access methods. If <replaceable class="parameter">pattern</replaceable> is specified, only access - methods whose names match the pattern are shown. If - <literal>+</literal> is appended to the command name, each access + methods whose names match the pattern are shown. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, each access method is listed with its associated handler function and description. </para> </listitem> @@ -1357,7 +1380,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dac"> <term> - <literal>\dAc[+] + <literal>\dAc[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">input-type-pattern</replaceable></link>]] </literal> @@ -1372,6 +1395,8 @@ SELECT $1 \parse stmt1 If <replaceable class="parameter">input-type-pattern</replaceable> is specified, only operator classes associated with input types whose names match that pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each operator class is listed with its associated operator family and owner. </para> @@ -1380,7 +1405,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-daf"> <term> - <literal>\dAf[+] + <literal>\dAf[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">input-type-pattern</replaceable></link>]] </literal> @@ -1395,6 +1420,8 @@ SELECT $1 \parse stmt1 If <replaceable class="parameter">input-type-pattern</replaceable> is specified, only operator families associated with input types whose names match that pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each operator family is listed with its owner. </para> @@ -1403,7 +1430,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dao"> <term> - <literal>\dAo[+] + <literal>\dAo[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">operator-family-pattern</replaceable></link>]] </literal> @@ -1419,6 +1446,8 @@ SELECT $1 \parse stmt1 If <replaceable class="parameter">operator-family-pattern</replaceable> is specified, only members of operator families whose names match that pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each operator is listed with its sort operator family (if it is an ordering operator). </para> @@ -1427,7 +1456,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dap"> <term> - <literal>\dAp[+] + <literal>\dAp[x+] [<link linkend="app-psql-patterns"><replaceable class="parameter">access-method-pattern</replaceable></link> [<link linkend="app-psql-patterns"><replaceable class="parameter">operator-family-pattern</replaceable></link>]] </literal> @@ -1442,6 +1471,8 @@ SELECT $1 \parse stmt1 If <replaceable class="parameter">operator-family-pattern</replaceable> is specified, only functions of operator families whose names match that pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, functions are displayed verbosely, with their actual parameter lists. </para> @@ -1449,13 +1480,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-db"> - <term><literal>\db[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\db[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists tablespaces. If <replaceable class="parameter">pattern</replaceable> is specified, only tablespaces whose names match the pattern are shown. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each tablespace is listed with its associated options, on-disk size, permissions and description. @@ -1465,7 +1498,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dc-lc"> - <term><literal>\dc[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dc[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists conversions between character-set encodings. @@ -1475,6 +1508,8 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each object is listed with its associated description. </para> @@ -1483,7 +1518,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dconfig"> - <term><literal>\dconfig[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dconfig[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists server configuration parameters and their values. @@ -1492,6 +1527,8 @@ SELECT $1 \parse stmt1 a <replaceable class="parameter">pattern</replaceable>, only parameters that are set to non-default values are listed. (Use <literal>\dconfig *</literal> to see all parameters.) + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each parameter is listed with its data type, context in which the parameter can be set, and access privileges (if non-default access @@ -1502,13 +1539,15 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dc-uc"> - <term><literal>\dC[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dC[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists type casts. If <replaceable class="parameter">pattern</replaceable> is specified, only casts whose source or target types match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each object is listed with its associated description. </para> @@ -1517,7 +1556,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dd-lc"> - <term><literal>\dd[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dd[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Shows the descriptions of objects of type <literal>constraint</literal>, @@ -1534,6 +1573,8 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -1546,7 +1587,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dd-uc"> - <term><literal>\dD[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dD[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists domains. If <replaceable @@ -1555,6 +1596,8 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each object is listed with its associated permissions and description. </para> @@ -1563,7 +1606,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-ddp"> - <term><literal>\ddp [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\ddp[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists default access privilege settings. An entry is shown for @@ -1572,6 +1615,8 @@ SELECT $1 \parse stmt1 If <replaceable class="parameter">pattern</replaceable> is specified, only entries whose role name or schema name matches the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -1585,12 +1630,12 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-de"> - <term><literal>\dE[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\di[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dm[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\ds[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dt[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> - <term><literal>\dv[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dE[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\di[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dm[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\ds[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dt[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dv[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1603,7 +1648,10 @@ SELECT $1 \parse stmt1 You can specify any or all of these letters, in any order, to obtain a listing of objects of these types. For example, <literal>\dti</literal> lists - tables and indexes. If <literal>+</literal> is + tables and indexes. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, each object is listed with its persistence status (permanent, temporary, or unlogged), physical size on disk, and associated description if any. @@ -1618,14 +1666,17 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-des"> - <term><literal>\des[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\des[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign servers (mnemonic: <quote>external servers</quote>). If <replaceable class="parameter">pattern</replaceable> is specified, only those servers whose name matches the pattern - are listed. If the form <literal>\des+</literal> is used, a + are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, a full description of each server is shown, including the server's access privileges, type, version, options, and description. </para> @@ -1634,14 +1685,17 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-det"> - <term><literal>\det[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\det[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign tables (mnemonic: <quote>external tables</quote>). If <replaceable class="parameter">pattern</replaceable> is specified, only entries whose table name or schema name matches - the pattern are listed. If the form <literal>\det+</literal> - is used, generic options and the foreign table description + the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, + generic options and the foreign table description are also displayed. </para> </listitem> @@ -1649,15 +1703,18 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-deu"> - <term><literal>\deu[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\deu[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists user mappings (mnemonic: <quote>external users</quote>). If <replaceable class="parameter">pattern</replaceable> is specified, only those mappings whose user names match the - pattern are listed. If the form <literal>\deu+</literal> is - used, additional information about each mapping is shown. + pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, + additional information about each mapping is shown. </para> <caution> @@ -1672,15 +1729,18 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dew"> - <term><literal>\dew[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dew[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists foreign-data wrappers (mnemonic: <quote>external wrappers</quote>). If <replaceable class="parameter">pattern</replaceable> is specified, only those foreign-data wrappers whose name matches - the pattern are listed. If the form <literal>\dew+</literal> - is used, the access privileges, options, and description of the + the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, + the access privileges, options, and description of the foreign-data wrapper are also shown. </para> </listitem> @@ -1688,7 +1748,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-df-lc"> - <term><literal>\df[anptwS+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> ... ] ]</literal></term> + <term><literal>\df[anptwSx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> ... ] ]</literal></term> <listitem> <para> @@ -1709,7 +1769,9 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. - If the form <literal>\df+</literal> is used, additional information + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, additional information about each function is shown, including volatility, parallel safety, owner, security classification, access privileges, language, internal name (for C and internal functions only), @@ -1722,13 +1784,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-df-uc"> - <term><literal>\dF[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dF[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search configurations. If <replaceable class="parameter">pattern</replaceable> is specified, only configurations whose names match the pattern are shown. - If the form <literal>\dF+</literal> is used, a full description of + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, a full description of each configuration is shown, including the underlying text search parser and the dictionary list for each parser token type. </para> @@ -1736,13 +1800,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dfd"> - <term><literal>\dFd[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFd[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search dictionaries. If <replaceable class="parameter">pattern</replaceable> is specified, only dictionaries whose names match the pattern are shown. - If the form <literal>\dFd+</literal> is used, additional information + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, additional information is shown about each selected dictionary, including the underlying text search template and the option values. </para> @@ -1750,13 +1816,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dfp"> - <term><literal>\dFp[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFp[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search parsers. If <replaceable class="parameter">pattern</replaceable> is specified, only parsers whose names match the pattern are shown. - If the form <literal>\dFp+</literal> is used, a full description of + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, a full description of each parser is shown, including the underlying functions and the list of recognized token types. </para> @@ -1764,13 +1832,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dft"> - <term><literal>\dFt[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dFt[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists text search templates. If <replaceable class="parameter">pattern</replaceable> is specified, only templates whose names match the pattern are shown. - If the form <literal>\dFt+</literal> is used, additional information + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, additional information is shown about each template, including the underlying function names. </para> </listitem> @@ -1778,7 +1848,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dg"> - <term><literal>\dg[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dg[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists database roles. @@ -1789,7 +1859,9 @@ SELECT $1 \parse stmt1 <literal>S</literal> modifier to include system roles. If <replaceable class="parameter">pattern</replaceable> is specified, only those roles whose names match the pattern are listed. - If the form <literal>\dg+</literal> is used, additional information + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, additional information is shown about each role; currently this adds the comment for each role. </para> @@ -1798,11 +1870,13 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dl-lc"> - <term><literal>\dl[+]</literal></term> + <term><literal>\dl[x+]</literal></term> <listitem> <para> This is an alias for <command>\lo_list</command>, which shows a list of large objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each large object is listed with its associated permissions, if any. @@ -1811,7 +1885,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dl-uc"> - <term><literal>\dL[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dL[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists procedural languages. If <replaceable @@ -1819,7 +1893,10 @@ SELECT $1 \parse stmt1 is specified, only languages whose names match the pattern are listed. By default, only user-created languages are shown; supply the <literal>S</literal> modifier to include system - objects. If <literal>+</literal> is appended to the command name, each + objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, each language is listed with its call handler, validator, access privileges, and whether it is a system object. </para> @@ -1828,7 +1905,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dn"> - <term><literal>\dn[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dn[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> @@ -1837,6 +1914,8 @@ SELECT $1 \parse stmt1 is specified, only schemas whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each object is listed with its associated permissions and description, if any. </para> @@ -1845,7 +1924,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-do-lc"> - <term><literal>\do[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> [ <replaceable class="parameter">arg_pattern</replaceable> ] ] ]</literal></term> + <term><literal>\do[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> [ <replaceable class="parameter">arg_pattern</replaceable> [ <replaceable class="parameter">arg_pattern</replaceable> ] ] ]</literal></term> <listitem> <para> Lists operators with their operand and result types. @@ -1861,6 +1940,8 @@ SELECT $1 \parse stmt1 By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, additional information about each operator is shown, currently just the name of the underlying function. @@ -1870,7 +1951,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-do-uc"> - <term><literal>\dO[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dO[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists collations. @@ -1878,7 +1959,10 @@ SELECT $1 \parse stmt1 specified, only collations whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to - include system objects. If <literal>+</literal> is appended + include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, each collation is listed with its associated description, if any. Note that only collations usable with the current database's encoding @@ -1890,7 +1974,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dp-lc"> - <term><literal>\dp[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dp[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists tables, views and sequences with their @@ -1900,6 +1984,8 @@ SELECT $1 \parse stmt1 pattern are listed. By default only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -1914,7 +2000,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-dp-uc"> - <term><literal>\dP[itn+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dP[itnx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists partitioned relations. @@ -1934,6 +2020,8 @@ SELECT $1 \parse stmt1 </para> <para> + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, the sum of the sizes of each relation's partitions is also displayed, along with the relation's description. @@ -1947,7 +2035,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-drds"> - <term><literal>\drds [ <link linkend="app-psql-patterns"><replaceable class="parameter">role-pattern</replaceable></link> [ <link linkend="app-psql-patterns"><replaceable class="parameter">database-pattern</replaceable></link> ] ]</literal></term> + <term><literal>\drds[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">role-pattern</replaceable></link> [ <link linkend="app-psql-patterns"><replaceable class="parameter">database-pattern</replaceable></link> ] ]</literal></term> <listitem> <para> Lists defined configuration settings. These settings can be @@ -1957,6 +2045,8 @@ SELECT $1 \parse stmt1 specific roles and databases to list, respectively. If omitted, or if <literal>*</literal> is specified, all settings are listed, including those not role-specific or database-specific, respectively. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -1970,7 +2060,7 @@ SELECT $1 \parse stmt1 <varlistentry id="app-psql-meta-command-drg"> - <term><literal>\drg[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\drg[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists information about each granted role membership, including @@ -1984,19 +2074,23 @@ SELECT $1 \parse stmt1 <literal>S</literal> modifier to include system roles. If <replaceable class="parameter">pattern</replaceable> is specified, only grants to those roles whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> </listitem> </varlistentry> <varlistentry id="app-psql-meta-command-drp"> - <term><literal>\dRp[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dRp[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists replication publications. If <replaceable class="parameter">pattern</replaceable> is specified, only those publications whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, the tables and schemas associated with each publication are shown as well. </para> @@ -2004,13 +2098,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-drs"> - <term><literal>\dRs[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dRs[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists replication subscriptions. If <replaceable class="parameter">pattern</replaceable> is specified, only those subscriptions whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, additional properties of the subscriptions are shown. </para> @@ -2018,12 +2114,14 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dt"> - <term><literal>\dT[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dT[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists data types. If <replaceable class="parameter">pattern</replaceable> is specified, only types whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each type is listed with its internal name and size, its allowed values if it is an <type>enum</type> type, and its associated permissions. @@ -2035,7 +2133,7 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-du"> - <term><literal>\du[S+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\du[Sx+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists database roles. @@ -2046,7 +2144,9 @@ SELECT $1 \parse stmt1 <literal>S</literal> modifier to include system roles. If <replaceable class="parameter">pattern</replaceable> is specified, only those roles whose names match the pattern are listed. - If the form <literal>\du+</literal> is used, additional information + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, additional information is shown about each role; currently this adds the comment for each role. </para> @@ -2054,27 +2154,31 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dx-lc"> - <term><literal>\dx[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dx[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists installed extensions. If <replaceable class="parameter">pattern</replaceable> is specified, only those extensions whose names match the pattern are listed. - If the form <literal>\dx+</literal> is used, all the objects belonging + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. + If <literal>+</literal> is appended to the command name, all the objects belonging to each matching extension are listed. </para> </listitem> </varlistentry> <varlistentry id="app-psql-meta-command-dx-uc"> - <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dX[x] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists extended statistics. If <replaceable class="parameter">pattern</replaceable> is specified, only those extended statistics whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -2090,13 +2194,15 @@ SELECT $1 \parse stmt1 </varlistentry> <varlistentry id="app-psql-meta-command-dy"> - <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\dy[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists event triggers. If <replaceable class="parameter">pattern</replaceable> is specified, only those event triggers whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, each object is listed with its associated description. </para> @@ -2684,13 +2790,15 @@ SELECT <varlistentry id="app-psql-meta-command-list"> - <term><literal>\l[+]</literal> or <literal>\list[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\l[x+]</literal> or <literal>\list[x+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> List the databases in the server and show their names, owners, character set encodings, and access privileges. If <replaceable class="parameter">pattern</replaceable> is specified, only databases whose names match the pattern are listed. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. If <literal>+</literal> is appended to the command name, database sizes, default tablespaces, and descriptions are also displayed. (Size information is only available for databases that the current @@ -3693,7 +3801,7 @@ testdb=> <userinput>\setenv LESS -imx <varlistentry id="app-psql-meta-command-z"> - <term><literal>\z[S] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> + <term><literal>\z[Sx] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term> <listitem> <para> Lists tables, views and sequences with their @@ -3703,6 +3811,8 @@ testdb=> <userinput>\setenv LESS -imx pattern are listed. By default only user-created objects are shown; supply a pattern or the <literal>S</literal> modifier to include system objects. + If <literal>x</literal> is appended to the command name, the results + are displayed in expanded mode. </para> <para> @@ -5386,6 +5496,30 @@ testdb=> <userinput>\df int*pl * bigi </programlisting> </para> + <para> + Here, the <literal>+</literal> option is used to display additional + information about one of these functions, and <literal>x</literal> is used + to display the results in expanded mode: +<programlisting> +testdb=> <userinput>\df+x int4*pl * bigint</userinput> +List of functions +-[ RECORD 1 ]-------+----------------------------- +Schema | pg_catalog +Name | int48pl +Result data type | bigint +Argument data types | integer, bigint +Type | func +Volatility | immutable +Parallel | safe +Owner | pgdevel +Security | invoker +Access privileges | +Language | internal +Internal name | int48pl +Description | implementation of + operator +</programlisting> + </para> + <para> When suitable, query results can be shown in a crosstab representation with the <command>\crosstabview</command> command: diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c new file mode 100644 index 5dd4c2d..8eda1fd --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -377,7 +377,10 @@ exec_command(const char *cmd, else if (strcmp(cmd, "if") == 0) status = exec_command_if(scan_state, cstack, query_buf); else if (strcmp(cmd, "l") == 0 || strcmp(cmd, "list") == 0 || - strcmp(cmd, "l+") == 0 || strcmp(cmd, "list+") == 0) + strcmp(cmd, "lx") == 0 || strcmp(cmd, "listx") == 0 || + strcmp(cmd, "l+") == 0 || strcmp(cmd, "list+") == 0 || + strcmp(cmd, "lx+") == 0 || strcmp(cmd, "listx+") == 0 || + strcmp(cmd, "l+x") == 0 || strcmp(cmd, "list+x") == 0) status = exec_command_list(scan_state, active_branch, cmd); else if (strncmp(cmd, "lo_", 3) == 0) status = exec_command_lo(scan_state, active_branch, cmd); @@ -424,7 +427,9 @@ exec_command(const char *cmd, query_buf, previous_buf); else if (strcmp(cmd, "x") == 0) status = exec_command_x(scan_state, active_branch); - else if (strcmp(cmd, "z") == 0 || strcmp(cmd, "zS") == 0) + else if (strcmp(cmd, "z") == 0 || + strcmp(cmd, "zS") == 0 || strcmp(cmd, "zx") == 0 || + strcmp(cmd, "zSx") == 0 || strcmp(cmd, "zxS") == 0) status = exec_command_z(scan_state, active_branch, cmd); else if (strcmp(cmd, "!") == 0) status = exec_command_shell_escape(scan_state, active_branch); @@ -850,6 +855,7 @@ exec_command_d(PsqlScanState scan_state, char *pattern; bool show_verbose, show_system; + unsigned short int save_expanded; /* We don't do SQLID reduction on the pattern yet */ pattern = psql_scan_slash_option(scan_state, @@ -858,6 +864,16 @@ exec_command_d(PsqlScanState scan_state, show_verbose = strchr(cmd, '+') ? true : false; show_system = strchr(cmd, 'S') ? true : false; + /* + * The 'x' option turns expanded mode on for this command only. This + * is allowed in all \d* commands, except \d by itself, since \dx is a + * separate command. So the 'x' option cannot appear immediately after + * \d, but it can appear after \d followed by other options. + */ + save_expanded = pset.popt.topt.expanded; + if (cmd[1] != '\0' && strchr(&cmd[2], 'x')) + pset.popt.topt.expanded = 1; + switch (cmd[1]) { case '\0': @@ -873,13 +889,14 @@ exec_command_d(PsqlScanState scan_state, { char *pattern2 = NULL; - if (pattern && cmd[2] != '\0' && cmd[2] != '+') + if (pattern && cmd[2] != '\0' && cmd[2] != '+' && cmd[2] != 'x') pattern2 = psql_scan_slash_option(scan_state, OT_NORMAL, NULL, true); switch (cmd[2]) { case '\0': case '+': + case 'x': success = describeAccessMethods(pattern, show_verbose); break; case 'c': @@ -941,6 +958,7 @@ exec_command_d(PsqlScanState scan_state, case 'p': case 't': case 'w': + case 'x': success = exec_command_dfo(scan_state, cmd, pattern, show_verbose, show_system); break; @@ -981,6 +999,7 @@ exec_command_d(PsqlScanState scan_state, case 't': case 'i': case 'n': + case 'x': success = listPartitionedTables(&cmd[2], pattern, show_verbose); break; default: @@ -1041,6 +1060,7 @@ exec_command_d(PsqlScanState scan_state, { case '\0': case '+': + case 'x': success = listTSConfigs(pattern, show_verbose); break; case 'p': @@ -1093,6 +1113,9 @@ exec_command_d(PsqlScanState scan_state, status = PSQL_CMD_UNKNOWN; } + /* Restore original expanded mode */ + pset.popt.topt.expanded = save_expanded; + free(pattern); } else @@ -2044,14 +2067,23 @@ exec_command_list(PsqlScanState scan_sta { char *pattern; bool show_verbose; + unsigned short int save_expanded; pattern = psql_scan_slash_option(scan_state, OT_NORMAL, NULL, true); show_verbose = strchr(cmd, '+') ? true : false; + /* if 'x' option specified, force expanded mode */ + save_expanded = pset.popt.topt.expanded; + if (strchr(cmd, 'x')) + pset.popt.topt.expanded = 1; + success = listAllDbs(pattern, show_verbose); + /* restore original expanded mode */ + pset.popt.topt.expanded = save_expanded; + free(pattern); } else @@ -3061,14 +3093,23 @@ exec_command_z(PsqlScanState scan_state, { char *pattern; bool show_system; + unsigned short int save_expanded; pattern = psql_scan_slash_option(scan_state, OT_NORMAL, NULL, true); show_system = strchr(cmd, 'S') ? true : false; + /* if 'x' option specified, force expanded mode */ + save_expanded = pset.popt.topt.expanded; + if (strchr(cmd, 'x')) + pset.popt.topt.expanded = 1; + success = permissionsList(pattern, show_system); + /* restore original expanded mode */ + pset.popt.topt.expanded = save_expanded; + free(pattern); } else diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c new file mode 100644 index d5543fd..9c2d522 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -309,9 +309,9 @@ describeFunctions(const char *functypes, /* No "Parallel" column before 9.6 */ static const bool translate_columns_pre_96[] = {false, false, false, false, true, true, false, true, false, false, false, false}; - if (strlen(functypes) != strspn(functypes, "anptwS+")) + if (strlen(functypes) != strspn(functypes, "anptwSx+")) { - pg_log_error("\\df only takes [anptwS+] as options"); + pg_log_error("\\df only takes [anptwSx+] as options"); return true; } diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c new file mode 100644 index fda8346..837651e --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -219,60 +219,60 @@ slashUsage(unsigned short int pager) HELP0("\n"); HELP0("Informational\n"); - HELP0(" (options: S = show system objects, + = additional detail)\n"); - HELP0(" \\d[S+] list tables, views, and sequences\n"); - HELP0(" \\d[S+] NAME describe table, view, sequence, or index\n"); - HELP0(" \\da[S] [PATTERN] list aggregates\n"); - HELP0(" \\dA[+] [PATTERN] list access methods\n"); - HELP0(" \\dAc[+] [AMPTRN [TYPEPTRN]] list operator classes\n"); - HELP0(" \\dAf[+] [AMPTRN [TYPEPTRN]] list operator families\n"); - HELP0(" \\dAo[+] [AMPTRN [OPFPTRN]] list operators of operator families\n"); - HELP0(" \\dAp[+] [AMPTRN [OPFPTRN]] list support functions of operator families\n"); - HELP0(" \\db[+] [PATTERN] list tablespaces\n"); - HELP0(" \\dc[S+] [PATTERN] list conversions\n"); - HELP0(" \\dconfig[+] [PATTERN] list configuration parameters\n"); - HELP0(" \\dC[+] [PATTERN] list casts\n"); - HELP0(" \\dd[S] [PATTERN] show object descriptions not displayed elsewhere\n"); - HELP0(" \\dD[S+] [PATTERN] list domains\n"); - HELP0(" \\ddp [PATTERN] list default privileges\n"); - HELP0(" \\dE[S+] [PATTERN] list foreign tables\n"); - HELP0(" \\des[+] [PATTERN] list foreign servers\n"); - HELP0(" \\det[+] [PATTERN] list foreign tables\n"); - HELP0(" \\deu[+] [PATTERN] list user mappings\n"); - HELP0(" \\dew[+] [PATTERN] list foreign-data wrappers\n"); - HELP0(" \\df[anptw][S+] [FUNCPTRN [TYPEPTRN ...]]\n" + HELP0(" (options: S = show system objects, x = expanded mode, + = additional detail)\n"); + HELP0(" \\d[Sx+] list tables, views, and sequences\n"); + HELP0(" \\d[S+] NAME describe table, view, sequence, or index\n"); + HELP0(" \\da[Sx] [PATTERN] list aggregates\n"); + HELP0(" \\dA[x+] [PATTERN] list access methods\n"); + HELP0(" \\dAc[x+] [AMPTRN [TYPEPTRN]] list operator classes\n"); + HELP0(" \\dAf[x+] [AMPTRN [TYPEPTRN]] list operator families\n"); + HELP0(" \\dAo[x+] [AMPTRN [OPFPTRN]] list operators of operator families\n"); + HELP0(" \\dAp[x+] [AMPTRN [OPFPTRN]] list support functions of operator families\n"); + HELP0(" \\db[x+] [PATTERN] list tablespaces\n"); + HELP0(" \\dc[Sx+] [PATTERN] list conversions\n"); + HELP0(" \\dconfig[x+] [PATTERN] list configuration parameters\n"); + HELP0(" \\dC[x+] [PATTERN] list casts\n"); + HELP0(" \\dd[Sx] [PATTERN] show object descriptions not displayed elsewhere\n"); + HELP0(" \\dD[Sx+] [PATTERN] list domains\n"); + HELP0(" \\ddp[x] [PATTERN] list default privileges\n"); + HELP0(" \\dE[Sx+] [PATTERN] list foreign tables\n"); + HELP0(" \\des[x+] [PATTERN] list foreign servers\n"); + HELP0(" \\det[x+] [PATTERN] list foreign tables\n"); + HELP0(" \\deu[x+] [PATTERN] list user mappings\n"); + HELP0(" \\dew[x+] [PATTERN] list foreign-data wrappers\n"); + HELP0(" \\df[anptw][Sx+] [FUNCPTRN [TYPEPTRN ...]]\n" " list [only agg/normal/procedure/trigger/window] functions\n"); - HELP0(" \\dF[+] [PATTERN] list text search configurations\n"); - HELP0(" \\dFd[+] [PATTERN] list text search dictionaries\n"); - HELP0(" \\dFp[+] [PATTERN] list text search parsers\n"); - HELP0(" \\dFt[+] [PATTERN] list text search templates\n"); - HELP0(" \\dg[S+] [PATTERN] list roles\n"); - HELP0(" \\di[S+] [PATTERN] list indexes\n"); - HELP0(" \\dl[+] list large objects, same as \\lo_list\n"); - HELP0(" \\dL[S+] [PATTERN] list procedural languages\n"); - HELP0(" \\dm[S+] [PATTERN] list materialized views\n"); - HELP0(" \\dn[S+] [PATTERN] list schemas\n"); - HELP0(" \\do[S+] [OPPTRN [TYPEPTRN [TYPEPTRN]]]\n" + HELP0(" \\dF[x+] [PATTERN] list text search configurations\n"); + HELP0(" \\dFd[x+] [PATTERN] list text search dictionaries\n"); + HELP0(" \\dFp[x+] [PATTERN] list text search parsers\n"); + HELP0(" \\dFt[x+] [PATTERN] list text search templates\n"); + HELP0(" \\dg[Sx+] [PATTERN] list roles\n"); + HELP0(" \\di[Sx+] [PATTERN] list indexes\n"); + HELP0(" \\dl[x+] list large objects, same as \\lo_list\n"); + HELP0(" \\dL[Sx+] [PATTERN] list procedural languages\n"); + HELP0(" \\dm[Sx+] [PATTERN] list materialized views\n"); + HELP0(" \\dn[Sx+] [PATTERN] list schemas\n"); + HELP0(" \\do[Sx+] [OPPTRN [TYPEPTRN [TYPEPTRN]]]\n" " list operators\n"); - HELP0(" \\dO[S+] [PATTERN] list collations\n"); - HELP0(" \\dp[S] [PATTERN] list table, view, and sequence access privileges\n"); - HELP0(" \\dP[itn+] [PATTERN] list [only index/table] partitioned relations [n=nested]\n"); - HELP0(" \\drds [ROLEPTRN [DBPTRN]] list per-database role settings\n"); - HELP0(" \\drg[S] [PATTERN] list role grants\n"); - HELP0(" \\dRp[+] [PATTERN] list replication publications\n"); - HELP0(" \\dRs[+] [PATTERN] list replication subscriptions\n"); - HELP0(" \\ds[S+] [PATTERN] list sequences\n"); - HELP0(" \\dt[S+] [PATTERN] list tables\n"); - HELP0(" \\dT[S+] [PATTERN] list data types\n"); - HELP0(" \\du[S+] [PATTERN] list roles\n"); - HELP0(" \\dv[S+] [PATTERN] list views\n"); - HELP0(" \\dx[+] [PATTERN] list extensions\n"); - HELP0(" \\dX [PATTERN] list extended statistics\n"); - HELP0(" \\dy[+] [PATTERN] list event triggers\n"); - HELP0(" \\l[+] [PATTERN] list databases\n"); - HELP0(" \\sf[+] FUNCNAME show a function's definition\n"); - HELP0(" \\sv[+] VIEWNAME show a view's definition\n"); - HELP0(" \\z[S] [PATTERN] same as \\dp\n"); + HELP0(" \\dO[Sx+] [PATTERN] list collations\n"); + HELP0(" \\dp[Sx] [PATTERN] list table, view, and sequence access privileges\n"); + HELP0(" \\dP[itnx+] [PATTERN] list [only index/table] partitioned relations [n=nested]\n"); + HELP0(" \\drds[x] [ROLEPTRN [DBPTRN]] list per-database role settings\n"); + HELP0(" \\drg[Sx] [PATTERN] list role grants\n"); + HELP0(" \\dRp[x+] [PATTERN] list replication publications\n"); + HELP0(" \\dRs[x+] [PATTERN] list replication subscriptions\n"); + HELP0(" \\ds[Sx+] [PATTERN] list sequences\n"); + HELP0(" \\dt[Sx+] [PATTERN] list tables\n"); + HELP0(" \\dT[Sx+] [PATTERN] list data types\n"); + HELP0(" \\du[Sx+] [PATTERN] list roles\n"); + HELP0(" \\dv[Sx+] [PATTERN] list views\n"); + HELP0(" \\dx[x+] [PATTERN] list extensions\n"); + HELP0(" \\dX[x] [PATTERN] list extended statistics\n"); + HELP0(" \\dy[x+] [PATTERN] list event triggers\n"); + HELP0(" \\l[x+] [PATTERN] list databases\n"); + HELP0(" \\sf[+] FUNCNAME show a function's definition\n"); + HELP0(" \\sv[+] VIEWNAME show a view's definition\n"); + HELP0(" \\z[Sx] [PATTERN] same as \\dp\n"); HELP0("\n"); HELP0("Large Objects\n"); diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out new file mode 100644 index 36dc31c..e60db50 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -2841,6 +2841,19 @@ Owned by: public.psql_serial_tab.id pg_catalog | exp | double precision | double precision | func pg_catalog | exp | numeric | numeric | func +\dfx exp +Schema | pg_catalog +Name | exp +Result data type | double precision +Argument data types | double precision +Type | func +--------------------+----------------- +Schema | pg_catalog +Name | exp +Result data type | numeric +Argument data types | numeric +Type | func + \pset tuples_only false \pset expanded on \d psql_serial_tab_id_seq @@ -3047,6 +3060,49 @@ Access method: heap tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes | (4 rows) +-- \d with 'x' enables expanded mode, but only without a pattern +\d+x tbl_heap + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+----------------+-----------+----------+---------+----------+--------------+------------- + f1 | integer | | | | plain | | + f2 | character(100) | | | | extended | | + +\d+x +List of relations +-[ RECORD 1 ]--------------------- +Schema | tableam_display +Name | mat_view_heap_psql +Type | materialized view +Owner | regress_display_role +Persistence | permanent +Size | 0 bytes +Description | +-[ RECORD 2 ]--------------------- +Schema | tableam_display +Name | tbl_heap +Type | table +Owner | regress_display_role +Persistence | permanent +Size | 0 bytes +Description | +-[ RECORD 3 ]--------------------- +Schema | tableam_display +Name | tbl_heap_psql +Type | table +Owner | regress_display_role +Persistence | permanent +Size | 0 bytes +Description | +-[ RECORD 4 ]--------------------- +Schema | tableam_display +Name | view_heap_psql +Type | view +Owner | regress_display_role +Persistence | permanent +Size | 0 bytes +Description | + RESET ROLE; RESET search_path; DROP SCHEMA tableam_display CASCADE; @@ -5233,6 +5289,30 @@ List of access methods btree | float_ops | real | double precision | 3 | in_range(real,real,double precision,boolean,boolean) (8 rows) +\dApx+ btree time_ops +List of support functions of operator families +-[ RECORD 1 ]---------+--------------------------------------------------------------------------------- +AM | btree +Operator family | time_ops +Registered left type | time without time zone +Registered right type | time without time zone +Number | 1 +Function | time_cmp(time without time zone,time without time zone) +-[ RECORD 2 ]---------+--------------------------------------------------------------------------------- +AM | btree +Operator family | time_ops +Registered left type | time without time zone +Registered right type | time without time zone +Number | 4 +Function | btequalimage(oid) +-[ RECORD 3 ]---------+--------------------------------------------------------------------------------- +AM | btree +Operator family | time_ops +Registered left type | time without time zone +Registered right type | interval +Number | 3 +Function | in_range(time without time zone,time without time zone,interval,boolean,boolean) + \dAp * pg_catalog.uuid_ops List of support functions of operator families AM | Operator family | Registered left type | Registered right type | Number | Function @@ -6826,5 +6906,15 @@ CREATE TABLE defprivs (a int); public | defprivs | table | (default) | | (1 row) +\zx defprivs +Access privileges +-[ RECORD 1 ]-----+---------- +Schema | public +Name | defprivs +Type | table +Access privileges | (default) +Column privileges | +Policies | + \pset null '' DROP TABLE defprivs; diff --git a/src/test/regress/sql/psql.sql b/src/test/regress/sql/psql.sql new file mode 100644 index c5021fc..2bc7f95 --- a/src/test/regress/sql/psql.sql +++ b/src/test/regress/sql/psql.sql @@ -498,6 +498,7 @@ create table psql_serial_tab (id serial) \d psql_serial_tab_id_seq \pset tuples_only true \df exp +\dfx exp \pset tuples_only false \pset expanded on \d psql_serial_tab_id_seq @@ -560,6 +561,9 @@ CREATE MATERIALIZED VIEW mat_view_heap_p \dv+ \set HIDE_TABLEAM on \d+ +-- \d with 'x' enables expanded mode, but only without a pattern +\d+x tbl_heap +\d+x RESET ROLE; RESET search_path; DROP SCHEMA tableam_display CASCADE; @@ -1309,6 +1313,7 @@ drop role regress_partitioning_role; \dAo+ btree float_ops \dAo * pg_catalog.jsonb_path_ops \dAp+ btree float_ops +\dApx+ btree time_ops \dAp * pg_catalog.uuid_ops -- check \dconfig @@ -1927,5 +1932,6 @@ ROLLBACK; CREATE TABLE defprivs (a int); \pset null '(default)' \z defprivs +\zx defprivs \pset null '' DROP TABLE defprivs; ^ permalink raw reply [nested|flat] 123+ messages in thread
* Re: psql: Option to use expanded mode for various meta-commands 2024-12-30 09:10 psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2024-12-30 15:48 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 2025-01-08 13:43 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> @ 2025-01-09 03:18 ` Greg Sabino Mullane <[email protected]> 2025-01-14 17:10 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 0 siblings, 1 reply; 123+ messages in thread From: Greg Sabino Mullane @ 2025-01-09 03:18 UTC (permalink / raw) To: Dean Rasheed <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Wed, Jan 8, 2025 at 8:44 AM Dean Rasheed <[email protected]> wrote: > Attached is a more complete patch +1, looks good So in the end, I decided to just add a sentence to each command's > description, keeping it as > short as possible. > Yes, that makes sense. Cheers, Greg ^ permalink raw reply [nested|flat] 123+ messages in thread
* Re: psql: Option to use expanded mode for various meta-commands 2024-12-30 09:10 psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2024-12-30 15:48 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 2025-01-08 13:43 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2025-01-09 03:18 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> @ 2025-01-14 17:10 ` Dean Rasheed <[email protected]> 0 siblings, 0 replies; 123+ messages in thread From: Dean Rasheed @ 2025-01-14 17:10 UTC (permalink / raw) To: Greg Sabino Mullane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Thu, 9 Jan 2025 at 03:18, Greg Sabino Mullane <[email protected]> wrote: > > On Wed, Jan 8, 2025 at 8:44 AM Dean Rasheed <[email protected]> wrote: >> >> Attached is a more complete patch > > +1, looks good > Thanks for looking. I've pushed this now. (I realised that I had missed \lo_list, so I added support for that too, since it would have been odd to allow "x" in \dl, but not \lo_list.) Regards, Dean ^ permalink raw reply [nested|flat] 123+ messages in thread
end of thread, other threads:[~2025-01-14 17:10 UTC | newest] Thread overview: 123+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-09-08 09:54 [PATCH v21 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH 02/12] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2024-12-30 09:10 psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2024-12-30 15:48 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 2025-01-08 13:43 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]> 2025-01-09 03:18 ` Re: psql: Option to use expanded mode for various meta-commands Greg Sabino Mullane <[email protected]> 2025-01-14 17:10 ` Re: psql: Option to use expanded mode for various meta-commands Dean Rasheed <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox